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Description 

[0001] The present invention refers to video camera 
monitoring of escalators and/or moving wail<s according 
to tlie definition of the independent claims. 
[0002] Such monitoring systems are well known in dif- 
ferent embodiments as escalator start lock or escalator 
restart control. Through such monitoring systems the 
escalatorrestartafteravoluntary or erroneous actuation 
of the emergency stop or other safety devices must re- 
main blocked, until no person or object are present in 
the monitored field of the safety device. 
[0003] In particular, the escalator restart control con- 
ditions are established by the norm E115: an escalator, 
which is used to transport people in environments such 
as railway stations, shopping centres etc., should be 
monitored for security reasons. The monitoring is re- 
stricted to the case, where an escalator is stopped, and 
a safe restart is required. A safe restart may only be per- 
formed in situations where the escalator is repeatedly 
tested for emptiness, i.e. there are no persons or obsta- 
cles on the moving part and the entry regions of the es- 
calator. The required period of emptiness is typically ad- 
justed to 1 0 seconds. Over this period repeated checks 
for emptiness may be performed every 0.1 seconds. 
[0004] Monitoring systems of this type are for exam- 
ple described in EP 1013599, JP 10236757 and in JP 
10265163. 

[0005] EP 1 01 3599 discloses a monitoring system for 
escalator restart control, which detects the presence of 

persons or objects on the escalator through a set of 
cameras situated above the escalator. Practical exper- 
iments have demonstrated that this system does not 
work in the case of strong sun irradiation, faint diffused 
light and in the case of rain, drizzle and fog, and that 
underthese circumstances an unambiguous perception 
of the emptiness of the escalator cannot be assured. 
[0006] JP 10236757 shows a remote supervisory sys- 
tem whereby without dispatching a clerk in charge to a 
setting site of an escalator, a moving guide device is re- 
motely supervised, start and stop control can be per- 
formed. This remote supervisory system comprises ITV 
cameras supervising an escalator and its periphery and 
a central controller provided in a remote area to control 
a start/stop of the escalator. 

[0007] JP 10236757 shows an escalator controller to 

judge a phenomenon and to speedily correspond to a 
falling accident, etc., in accordance with apicked-up pic- 
ture image of an escalator and its periphery. 
[0008] Both JP 1 0236757 and JP 1 0236757 disclose 
monitoring systems, which do not work properly under 
special illumination conditions and cannot guarantee 
the unequivocal perception of the emptiness of the es- 
calator. In particular, shadows or dirty spots on the es- 
calator can be eventually confused with people or ob- 
jects lying on the escalator itself. 
[0009] The object of the present invention is to con- 
ceive a monitoring of obstacles and persons on escala- 



tors and/or moving walks, which allows a reliable and 
univocal detection of persons or obstacles lying in the 
monitored field of the escalator and/or moving walk. 
[001 0] According to the present invention this object 

5 is achieved by a monitoring system for the detection of 
obstacles and persons on escalators and/or moving 
walks comprising at least one video camera for the ac- 
quisition of stereoscopic images. 
[001 1 ] With the word stereoscopic images is meant 

10 a pair of pictures of the same field of view taken by two 
cameras situated at slightly different positions or taken 
by the same camera placed in two slightly different po- 
sitions, so that the same field of view is imaged under 
two slightly different angles. The objects on the escala- 

f5 tor, which should be detected, have the property of being 
closer to the camera than the escalator on which they 
are placed. 

The advantage of stereoscopic images is that these ob- 
jects appear at different positions in the pair of stereo- 
20 scopic images. Disturbances like dirt or inscriptions on 
the escalator appear on the same position in the pair of 
stereoscopic images, so that it is univocally possible to 
detect the presence of objects and persons on the es- 
calator. 

25 [0012] As obstacles or persons are understood ob- 
jects and bodies, whose dimensions are such as to en- 
danger the safe operation of the escalator and/or mov- 
ing walk. 

[0013] In preferred embodiments of the invention 
30 pairs of video cameras are located above the escalator 

or in the escalator balustrade. This embodiment exhibits 
the advantage that an optimal field of view of the esca- 
lator can be achieved, i.e. under a view angle of 45° the 
obstacles and persons are neither too close to the cam- 
35 era (too big in the image) nor too far away (too small in 
the image). 

If the escalator is very long, more than one pair of cam- 
eras is necessary to monitor conveniently the whole 
stroke of the escalator. 
40 [0014] In another preferred embodiment the monitor- 
ing system includes a processing unit to process the 
stereoscopic images. This embodiment exhibits the ad- 
vantage that the monitoring system can automatically 
process the acquired images and can autonomously 
45 come to the decision whether obstacles are present on 
the escalator or not. 

[0015] In this context with processing the stereo- 
scopic images is meant any operation preferably per- 
formed on digital images such as loading, storing, com- 
50 parison, differencing, rectifying, warping, reconstruc- 
tion, segmentation, grouping, edge detection, Hough 
transformations, extraction etc. as described below in 
the detailed description of the invention. 
[001 6] The processing unit can be a personal compu- 
55 ter or a standardised non-expensive processor integrat- 
ed in the camera or in any other part of the escalator 
equipment needing no special device to be mounted. 
[001 7] In another preferred embodiment the process- 
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ing unit and the canneras can be connected together by 
linking nneans and even with the escalator controller. 
This embodinnent exhibits the advantage that the mon- 
itoring system can automatically process the acquired 
images, can autonomously come to the decision wheth- 
er obstacles are present on the escalator or not and can 
finally restart automatically the escalator basing on the 
obtained information. With linking means is under- 
stood any physical means, such cables, signals or a da- 
ta exchange bus, which allows data to be exchanged 
and transmitted between two or more acquisition, 
processing and controlling units. 
[0018] According to the present invention the object 
is also achieved by a method for the detection of ob- 
stacles and/or persons on escalators and/or moving 
walks, whereby at least one video camera acquires ster- 
eoscopic images and a processing unit processes these 
images. The advantage of this method is that it is easy 
to perform and reliable. 

[0019] Another preferred embodiment of the invention 
refers to a computer program product for the detec- 
tion of obstacles and/or persons on escalators and/or 

moving walks, which loads in a processor and process- 
es stereoscopic images of the escalator and/or moving 
walk. The advantage of the computer program product 
is that it is loadable everywhere, locally or remotely in a 
central server and that an update is easy to perform. 
[0020] Preferred embodiments of the invention are 
described in detail below with reference to the following 
schematic drawings: 

Fig. 1 : A complete representation of the escalator 
equipped with the monitoring system according to 
the invention. 

Fig. 2: Monitoring system whereby a pair of camer- 
as is placed in the escalator balustrade. 

Fig. 3: Monitoring system whereby a pair of camer- 
as is mounted at the top of two posts placed along 
the escalator. 

Fig. 4: Collaboration diagram for image data ex- 
change using a shared memory. 

Fig. 5: Data flow diagram for the full system. 

[0021] Fig. 1 shows a complete representation of the 
escalator equipped with the monitoring system accord- 
ing to the invention. On the escalator 1 is standing a 
person 2, which is in the field of view of a pair of video 
cameras 3.1 and 3.2 placed at slightly different positions 
above the escalator. The cameras can therefore acquire 
pairs of stereoscopic images of the escalator. 
[0022] Image acquisition is performed using pairs of 
cameras, where the number nc of cameras required de- 
pends on the height of the staircase H. An estimate is 
given by nc = 4 + H, where H is the height of the esca- 



lator. For example, for a staircase spanning four meters 
in height four stereo camera pairs, i.e. eight cameras at 
all, are necessary. 

[0023] For the internal part of the escalator two cam- 

5 eras with a focal length of 6 millimetres (nominal) in 
turned mode, i.e. the vertical image extension is larger 
than the horizontal one, are suggested. The entry re- 
gions on the top and on the bottom of the escalator also 
require cameras with a focal length of 6 millimetres (also 
in turned mode). 

[0024] Environment and escalator parameters influ- 
encing the placement and the number of the cameras 
are for example the length of the escalator, which can 
be up to 1 00 meters. The escalator may be located in- 
or outdoors with or without covering. Escalator stairs 
may be coloured or hold inscriptions. An opaque object 
of cylindrical shape and minimum size of 0.1 5 meters in 
diameter and 0.15 meters in height must be detected as 
a necessary requirement. The illumination may vary 
over the escalator area, a minimum illumination is given 
as 50 Lux for indoor placement and 1 5 Lux for outdoor 
placement. 

[0025] Fig. 2 shows a preferred embodiment of the 
monitoring system whereby a pair of cameras is placed 
in the escalator balustrade. 

[0026] Fig. 3 shows a preferred embodiment of the 
monitoring system whereby a pair of cameras is mount- 
ed at the top of two posts placed along the escalator. 
[0027] B/W cameras and DFG/BW1 frame grabbers 
manufactured by The Imaging Source with a progres- 
sive scan CCD image sensor can be used. 
An important additional requirement are lenses control- 
lable by voltage. This requirement stems from the highly 
varying illumination conditions. 

The used lens is a Cosmicar lens, type H612ER, with a 
focal length f of 6 millimeters. The aperture opening is 
controllable from f/1 .2 to f/360 through variation of the 
control voltage in the range 1 .5 to 5 volts. The aperture 
is controlled using the NuDAQ 6208 multi-channel ana- 
logue output card. 

[0028] The cameras are connected through the link- 
ing means 4 (for example Hirose cables) to a process- 
ing unit 5, which processes the digitalized stereoscopic 
images taken by the video camera pair. Thanks to algo- 
rithms described below, the processing unit detects the 
presence or not of the person on the escalator. Detec- 
tion is based on differencing rectified stereo pair images, 
where a warping transform is to overlay the left onto the 
right image, and vice versa. The 3D camera positions 
are obtained through model based pose estimation and 
disparity is used to obtain the warping transform. 
[0029] In particular, the task is to detect objects on an 
escalator, which can be considered as a moving back- 
ground, under real-world illumination conditions. The 
suggested solution consists of model based back- 
ground reconstruction, perspective warping of one im- 
age to the other in a stereo setup, and the final detection 
of differences in an image pyramid. Specifically, we em- 
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ploy a model based staircase pose estimator based on 
grouping of line features by the use of geometric invar- 
iants. The detection is based on measuring absolute pix- 
el differences between unwarped and warped images. 
Image differences are represented in an image pyramid 
according to Peter J. Burt, Tsai-Hong Hong, and Azriel 
Rosenfeld, "Segmentation and Estimation of Image Re- 
gion Properties Through Cooperative Hierarchical Com- 
putation", IEEE Transactions on Systems, IVIan and Cy- 
bernetics, 11 (12):802-809, December 1981, and seg- 
mented into background (staircase) and foreground (ob- 
stacles) employing the algorithm suggested in M . Spann 
and R. Wilson, "A Quad-Tree Approach to Image Seg- 
mentation which Combines Statistical and Spatial Infor- 
mation", Pattern Recognition, 18(3/4):257-269, 1985. 
[0030] Image processing is performed on PC-class 
machines (Intel Pentium). The number of PC boxes can 
be higherthan one. In a preferred embodiment each PC 
box is responsible for two stereo pairs, i.e. is connected 
with four cameras. Each PC is equipped with one 
NuDAQ 6208 and two DFG/BW1 cards. 
[0031] The software is written in C-i-+ and runs under 
the Linux operating system. Efficient image, computer 
vision and matrix algebra algorithms are provided by the 
used Intel Performance Primitives Library. 
[0032] The main software components are: 

• Acquisition, including aperture control. 

• Calibration of camera and system (offline). 

• Monitoring, state estimation and detection (on- 
line). 

The detection part is the time-critical part performed 
at escalator service time (online), whereas the cal- 
ibration part is done beforehand, i.e. at escalator as- 
sembly time (offline). 

[0033] Acquisition performs two tasks: 

• Providing grabbed images at some negotiated 
shared memory. 

• Control of the aperture based on image properties, 
e.g. maximization of the information content in the 
staircase region of interest (ROI). 

[0034] Fig. 4 shows a collaboration diagram, which 
explains the communication between acquisition com- 
ponent and processes requiring images, i.e. the offline 
and online components. The basic principles for syn- 
chronisation and communication are: 

• Components are Unix processes. 

• Image data are exchanged using shared memories. 

• Synchronisation for shared memory access uses 
semaphores. 

• Signalling between processes uses message 
queues. 

[0035] For the calibration and monitoring part, there 



are seven main tasks identified: 

• Radial/tangential undistortion. 

• IVIotion segmentation and ROI identification. 
5 • Edge and line extraction. 

• Geometric matching, model/data line correspond- 
ence, pose estimation. 

• Disparity calculation, warping table setup. 

• Staircase state estimation. 

10 • Image warping, segmentation, connected compo- 
nents labelling, decision support. 

The undistortion task is required in the offline and online 
parts. The next first four components can be summa- 
15 rized as offline component, whereas the last two are the 
online component. 

[0036] Fig. 5 shows the data flow for the system, with 
the above mentioned components. The external data 
stores provide undistortion parameters from internal cal- 

20 ibration and a CAD model of the staircase, i.e. a list of 
points and lines. Output, which is the result of detection, 
goes to another external data store. The main compo- 
nents: acquisition, offline and online are grouped in 
shaded areas. Undistortion is applied to images gath- 

25 ered by both, the online and the offline component. 
[0037] As stated above, the main components of the 
presented system are the acquisition part, the offline (or 
calibration) part and the online (or detection) part. The 
most interesting subparts, i.e. geometric matching 

30 (establishing correspondences between 2D-Data and 
3D-Model) in the offline part and detection from stereo 
images in the online part, will be now discussed in some 
detail in the following. 

[0038] In model-based pose estimation parameters 

55 describing relative orientation and position, i.e. the ex- 
trinsic camera parameters, are found using corre- 
spondence between data and model. In our case, the 
data are 2D lines extracted from single images and the 
model is a 3D wireframe object. Nearly horizontal lines 

40 are derived from the image data using standard edge 
detection based on directional image gradient and 
Hough transform techniques. To establish correspond- 
ence between data and model lines for each image in 
the stereo pair, and furthermore, between the two stereo 

45 pairs, the following matching procedure is applied 
(grouping based on cross ratio). 
[0039] The first step in matching is to identify possible 
correspondences between data and model lines. Under 
perspective projection, ratios of ratios of lines and ratios 

50 of ratios of angles, the so called cross ratios, are invar- 
iant. We employ cross ratios to identify groups of four 
lines out of a larger set of possible lines. Such a group 
of 4 lines, which in our case is characterized by the cross 
ratio obtained for the intersection points with an approx- 

55 imately orthogonal line, serves as a matching candidate 
to the staircase pattern. The definition for the cross ratio 
for four points p1 p4 on a line is given as: 
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Cr(p1 ,...,p4) = [(x3-x1)(x4-x2)]/[(x3-x2)(x4-x1)], 

where the x1,...,x4 are the corresponding positions of 

each point on the line. 

[0040] The following strategy for selecting data lines 
which are good candidates for correspondence to model 
lines was enn ployed: 

• Calculate the theoretical cross ratio, e.g. for four 
equally spaced points on a line this is Crt = 4/3. 

• Detect a reasonable set L (of size N) of close to hor- 
izontal lines fronn the data. 

• Calculate intersection points of those lines with a 
close to vertical line. 

• Calculate all M=(^) four-element subsets of lines li 
cL,i=1,..., M. 

• Calculate all cross ratios ci corresponding to sets li. 

• Sort the li with respect to Ici-CrtI (in ascending or- 
der). 

Only a portion of the sorted groups, corresponding to 
those of lower distance to Crt, is input to the pose esti- 
mation step, which is described below (estimation of 
position and orientation). 

[0041] Corresponding groups of lines are input to a 
procedure similar to RANSAC as described in M.A. Fis- 
chler and R.C. Bolles, "Random Sample Con-census: A 
Paradigm for Model Fitting with Applications to Image 
Analysis and Automated Cartography", Communica- 
tions of the ACM, 24(6):381-395, 1981 . Grouping based 
on cross ratio delivers improved sampling for RANSAC 
and reduces the number of necessary iterations. The 
basic idea in RANSAC is, that RANSAC uses as small 
an initial data set as feasible and enlarges this set with 
consistent data when possible. The required number of 
random selections ns of samples with a size of s fea- 
tures is given by Fischler and Bolles as: 
ns = log (1-pc)/log (1-pi®), 

where pc is the probability, that at least one sample of 
s = 4 lines is free from outliers. The probability that any 
selected sample is an inlier is denoted by pi. In our case, 
due to the improved sampling based on cross ratio, we 
can safely assume a high pi, e.g. pi=0,8, and choosing 
pc=0,99, we obtain a number of necessary RANSAC it- 
erations as low as ns = 9. 

Verification of the pose is based on the procedure de- 
vised by David G. Lowe, "Fitting Parameterized 3-D 
Models to Images", IEEE Transactions on Pattern Anal- 
ysis and Machine Intelligence, 13(5):441-450, May 
1 991 . Lowe approaches the problem of derivation of ob- 
ject pose from a given set of l<nown correspondences 
between 3D-model lines and 2D image lines by lineari- 
zation of projection parameters and application of New- 
ton's method. The result of the pose estimation step are 
two transformations from world to camera coordinate 
system, i.e. three translational and three rotational pa- 
rameters for each camera. 



[0042] The detection from stereo images involves 
detector calibration, i.e. derivation of disparity and der- 
ivation of the two-dimensional warping transform, and 
the detection itself, i.e. warping of one image to the oth- 

5 er, differencing of warped and unwarped images and, 
finally, segmentation of the difference image in order to 
obtain a decision. 

[0043] The warping transform is found from the 
staircase model and the two world to camera coordinate 

10 system and projective transforms obtained by the pose 
estimation procedure mentioned above. A perspective 
warping transfonn provides us with two warping tables 
which contain the coordinate mapping for both coordi- 
nate directions in the image plane. The warping tables 

15 are calculated from disparity, which is accurately given 
due correspondence via the model, in straightf onward 
fashion. 

[0044] The main idea in detection of obstacles is to 

warp one image, e.g. the left image to the right one, and 

20 perform some comparison. The objects on the stair- 
case, which should be detected, have the property of 
being closer to the camera than the staircase on which 
they are placed. Therefore, objects in the image being 
warped appear at different positions than they appear 

25 in the unwarped image. On the other hand, disturbances 
like dirt or inscriptions on the staircase, appear on the 
same position in warped and unwarped image. 
[0045] To summarise, an extension of stereo based 
obstacle detection procedures to regularly structured 

30 and non-flat back-ground was employed. Grouping 
based on a cross ratio constraint improved RANSAC 
sampling. Pose estimation provides externally calibrat- 
ed cameras, which simplify and accelerate stereo 
processing and the object detection task which is per- 

35 formed using a pyramid based segmentation proce- 
dure. A high reliability of the approach was found ex- 
perimentally, i.e. a rate of omission of an obstacle in the 
order of magnitude of 1 percent, and a rate of false de- 
tection of an obstacle in the order of magnitude of 5 per- 

40 cents. Cylindrical objects down to a size of less than 1 5 
centimeters in height were detected reliably. 
[0046] The processing unit is connected through the 
control line 6 to the escalator controller 7 and can 
therefore control the restarting of the escalator after a 

45 stop in dependence on the detection of a person or ob- 
stacle on the escalator. 

[0047] Signal connections between the PCs and the 

escalator control use simple wires, through which sig- 
nals from the staircase control go to each PC and back. 
50 Signals from the PC to the control are combined in dis- 
junctive fashion, e.g. an object is detected if any PC sig- 
nals a detected object, etc. 

[0048] Three output signals are provided to the con- 
trol: 

55 

• Object detected. 

• Warning, e.g. camera problem. 

• Failure, i.e. system networking. 
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[0049] Additionally, the system should support a so- 
called test mode, where images are fed into the system 
from stored location and not from the cameras. There- 
fore, two input signals are necessary: 

• Staircase in standstill. 

• Test mode requested. 

Signalling between monitoring and staircase control is 
done using the digital input/output channels of the 
NuDAO 6208 multi-channel analogue output card. Be- 
sides the analogue output channels, the NuDAO 6208 
card provides four input and four output channels. 
[0050] The controller is connected through the motor 
supply line 8 to the escalator motor 9 and can therefore 
restart the motor or keep it in a still position. 
[0051] Furthertechnical requirements for the monitor- 
ing system are that the system may consist of two inde- 
pendent channels or control units. A watchdog function 
is required, i.e. the system may be continuously 
checked for availability. 

[0052] It is obvious for the man skilled in the art, that 
the disclosed system and method using pairs of stere- 
oscopic images can be also used to detect persons and 
objects in an elevator car or in a lobby in front of elevator 
doors. 



Claims 

1. Monitoring system for the detection of obstacles 
and persons comprising at least one video camera 
and escalators and/or moving walks, 
characterized in tliat, 

the monitoring system acquires stereoscopic imag- 
es. 

2. IVIonitoring system according to claim 1 , 
characterized in that, 

the video cameras are located above the escalator 
and/or moving walk. 

3. Monitoring system according to claim 1 , 
characterized in that, 

the video cameras are located in the balustrade of 
the escalator and/or moving walk. 

4. Monitoring system according to claims 1 to 3, 
characterized In that, 

more than one pair of video cameras are arranged 
along the escalator and/or moving walk in order to 
monitor conveniently the whole stroke of the esca- 
lator and/or moving walk. 

5. Monitoring system according to claims 1 to 4, 
characterized In that, 

the monitoring system comprises a processing unit 
processing the stereoscopic images. 



6. Monitoring system according to claims 1 to 5, 
characterized in that, 

the monitoring system comprises means for linking 
the video cameras with the processing unit, in par- 
5 ticular a data exchange bus, and/or means for stor- 
ing the stereoscopic images. 

7. Monitoring system according to claim 5 to 6, 
characterized in that, 

10 the processing unit is at least one personal compu- 
ter loaded with an image processing software pro- 
gram basing on the principle to compare the digital 
data of a model image with those of an actual ac- 
quired image. 

15 

8. Monitoring system according to claim 5 to 6, 
characterized In that, 

the processing unit is integrated in the cameras. 

20 9. Monitoring system according to claims 5 to 8, 
characterized In that, 

the monitoring system is connected electrically to 
an escalator and/or moving walk control, which re- 
starts the escalator and/or moving walk after a stop 
25 only when no obstacle and/or person are detected 
on the escalator and/or moving walk. 

10. Computer program product for the detection of ob- 
stacles and/or persons on escalators and/or moving 

30 walks, 

characterized in that, 

the computer program product loads in a processor 
and processes stereoscopic images of the escala- 
tor and/or moving walk. 

35 

11. Computer program product according to claim 10, 
characterized in that, 

the computer program product restarts the escala- 
tor and/or moving walk after a stop only when no 
40 obstacle and/or person are detected on the escala- 
tor and/or moving walk. 

12. Method for the detection of obstacles and persons 
on escalators and/or moving walks, 

45 characterized in that, 

at least one video camera acquires stereoscopic 
images and a processing unit processes these im- 
ages. 

50 13. Method according to claim 11 , 
characterized In that, 

the escalator and/or moving walk is restarted auto- 
matically after a stop only when no obstacle and/or 
person are detected on the escalator and/or moving 
55 walk. 
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